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/etc/xinetd.d/tftp with the following contents: 


service tftp 

{ 


disable = no 


socket_type 

= dgram 

protocol 

= udp 

wai t 

= yes 

user 

= root 
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m 


server 

server_args 

per_source 

cps 

flags 

#only_from 


= /usr/sbin/in.tftpd 
= -s /tftpboot -u tftp 
= 11 
= 100 2 
= IPv4 

= 153.90.196.30 


The answer to the last question is no. Unless you do really hairy and hard- 
to-debug things, only your program can free memory that it allocated. 

The others are really good questions, and the only way to know for 
sure is to try it with your code. Even though you may not be using the 
malloc family of functions, you might be making library calls that allo¬ 
cate memory and then omitting some of the calls required to free it. 


I’ve tried this with only_from both commented and uncommented. I’ve 
also made sure that the firewall trusts UDP and TCP on port 69.1 veri¬ 
fied that the contents of /etc/xinetd.conf are correct, and I verified that 
tfptd is running via chkconfig. I also verified that port 69 is available via 
netstat. I’ve tried running in.tftpd in standalone mode (server_args = -1). 

I’ve been working on this problem for three days and am getting 
nowhere. I’m something of a newbie to Linux, but I have asked more 
experienced folks for insight to no avail and have spent hours trying to 
find instances of this problem on the Internet, also to no avail. So, I’m 
hoping you folks can point me in the right direction. 


The good news is that you can build a version of your program that 
uses GC for all memory management, including memory allocated in 
library code, by “ hooking” it in to malloc. See Listing 1 in this article: 

www.linuxjournal.com/article/6679 for an example. 

Don Marti, dmarti@ssc.com 

Runlevel Editing 


In the April 2005 Best of Technical Support, in “Old Red Hat”, 
Timothy Hamlin suggests changing the /etc/inittab entry from: 


Todd Trotter, ishamt@esus.cs.montana.edu 

It seems as though you have done almost everything correctly. Some 
issues come to mind though. First, change the user to nobody on the 
file /etc/xinetd.d/tftp; otherwise, the in.tftpd dcemon runs as root, which 
is not safe. 

Second, make sure the lines: 


x:5:respawn:/etc/Xll/prefdm -nodaemon 
to: 

x:3:respawn:/etc/Xll/prefdm -nodaemon 

to suppress the X graphical login. I think he made an error here. His 
reply will launch X at runlevel 3. Instead change: 


tftp 69/tcp 

tftp 69/udp 

are not commented out in the /etc!services file. Also, I suggest checking 
the file /etc/hosts,deny to see if you are blocking requests for the in.tftpd 
dcemon, for all services or for requests from a specific IP (client machine). 

For testing purposes only, make sure this file is empty, reload xinetd 
(service xinetd reload) and try again. Also, for testing only, turn off 
your firewall (service iptables stop) and test again. Test and make 
your setup work locally by issuing tftp localhost before testing 
remotely. Hope this helps. 

Felipe Barousse Boue, fbarousse@piensa.com 

Is Garbage Collection the Answer? 


I learned about garbage collection (GC) from your journal. I do have a 
problem. Let me explain the situation that exists. Initially, the project 
occupies 192MB of RAM in Linux. It was allowed to run continuous¬ 
ly. Then, after 12 hours, we noticed it was using 335MB. What is the 
solution for this problem? Is it due to garbage? Will the BDW garbage 
collector provide a solution? The project includes char pointers, and it 
doesn’t include any malloc functions. 

Will BDW GC work only if we include malloc, calloc or realloc func¬ 
tions? Can I have a program that runs along with my project and 
releases free memory? 

MythilyJ., mattuvar@yahoo.co.in 


id : 5 : initdefault: 

to: 

id : 3 : initdefault: 
to change the default runlevel. 

Also, in “Tweaking inodes and Block Sizes”, Don Marti points out 
that Red Hat 9 is no longer supported and that this might be an 
issue for an older 486 system. The bigger issue is the amount of 
RAM Red Hat requires for the install. I’m not sure if it will install 
with 32MB of RAM. It definitely won’t with 16MB, which is what 
my old 486 laptop had. 

Roland Roberts, roland@astrofoto.org 

Either inittab change will work. The second has the advantage of pre¬ 
serving the “runlevel 5 is GUI login” tradition that Red Hat users are 
used to. The Fedora release notes at fedora.redhat.com/docs/ 
release-notes/fc3/x86 list a Pentium as the minimum processor 
and 64MB as minimum memory for a text install. (See the last letter 
for an alternate approach.) 

Don Marti, dmarti@ssc.com 

What about Fedora Legacy? 


In the April 2005 Best of Technical Support, Don Marti writes that 
“Neither Red Hat 9 nor Red Hat 6.2 is still supported, which means 
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no more security updates.” Although Red Hat has dropped support 
for Red Hat 9, the community-based Fedora-Legacy Project 
(www.fedoralegacy.org) is working to provide security updates 
for Red Hat 9 as well as Red Hat 7.3 and Fedora Core 1 and (soon) 
2. Mr Marti does the project a disservice by ignoring its efforts. 

John Dalbec, jdalbec@cboss.com 

At the time we went to press, Fedora Legacy was not actively releas¬ 
ing security updates. 

Don Marti, dmarti@ssc.com 

Really, Fedora on a Pentium? 


The Best of Technical Support column in the April 2005 issue of LJ con¬ 
tains some incorrect and incomplete statements in response to a user who 
wants to use Red Hat 9 on 486 computers. Don Marti writes, “[Red Hat’s] 
successor, Fedora, requires a Pentium or better...No matter what you 
install, this class of machines will be too slow for a modem desktop.” The 
RULE Project (www.rule-project.org) proves this wrong. One year 
ago, I ran Red Hat 9 on a Pentium I laptop with 32MB of RAM. 
Thanks to it, I used KOffice to make a presentation and Firefox for 
home banking: www.rule-project.org/article.php3?id_article=55 
(see the linked screenshot). 

Less than one month ago, we announced a version of our installer for 
Fedora Core 3: www.rule-project.org/breve.php3?id_breve=19. 

Now, it certainly is true that full-fledged KDE, GNOME or 
OpenOffice.org installations under any desktop can be painfully 
slow, even on much newer computers. It is equally true that video 
editing or 3-D gaming requires state-of-the-art hardware. But, if by 
modern desktop, one means modern SOHO functionality—IMAP, 
digital signatures, HTML4/CSS support, CUPS, IM, Bayesian spam 
filtering, regardless of eye candy—there is no need to spend 
money. All it takes is a project such as RULE and efforts made on 
things such as mini-KDE. In any case, it is possible to run a mod¬ 
ern, mainstream distro on slow hardware, with a bit of care and the 
right approach to the problem. 

Marco Fioretti, mfioretti@mclink.it0 


Many on-line help resources are available on the Linux Journal 
Web pages. Sunsite mirror sites, FAQs and HOWTOs can all be 
found at www.linuxjournal.com. 

Answers published in Best of Technical Support are provided by 
a team of Linux experts. If you would like to submit a question 
for consideration for use in this column, please fill out the Web 
form at www.linuxjournal.com/lj-issues/techsup.html or send 
e-mail with the subject line "BTS" to bts@ssc.com. 

Please be sure to include your distribution, kernel version, any 
details that seem relevant and a full description of the problem. 
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www.monarchcomputer.com 

Cyclades Corporation C2, 1, 11 

www.cyclades.com 

Penguin Computing 19 

www.penguincomputing.com 

Emac, Inc. 87 

www.emacinc.com 

The Portland Group 36, 37 

www.pgroup.com 

EmperorLinux 15 

www.emperorlinux.com 

Rackspace Managed Hosting 5 

www.rackspace.com 

Etnus 51 

www.etnus.com 

SBE, Inc. 13 

www.sbei.com 

FairCom Corporation 7 

www.faircom.com 

Servers Direct 39 

www.serversdirect.com 

Free Software Foundation 69 

www.gnupress.org 

Star Micronics 61 

www.starmicronics.com 

Geek Cruises 41 

www.geekcruises.com 

Technologic Systems 45 

www.embeddedx86.com 

Google 47 

www.google.com/lj 

TUX MAGAZINE 68 

www.tuxmagazine.com 

Hurricane Electric 57 

www.he.net 

Tyan Computer USA 35 

www.tyan.com 

Iron Systems 85 

www.ironsystems.com 

ZT Group International 33 

www.ztgroup.com 

Java One 77 

java.sun.com/javaone/sf/pavilion/index.jsp 
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SUSE Linux Professional 9.3 


OPTion 


Novell released SUSE Linux Professional 9.3, 
which includes a complete Linux OS, more 
than 3,000 open-source packages and hun¬ 
dreds of open-source applications, productivi¬ 
ty software and home networking capabilities. 
Designed for both Linux newcomers and 
longtime users, SUSE Pro 9.3 offers many 
new features, including an OS built on kernel 
version 2.6.11, KDE 3.4 and GNOME 2.10, 
Firefox 1.0, OpenOffice.org 2.0, F-Spot photo 
organizer, The GIMP 2.2, Mono 1.1.4, 
KDevelop 3.2, Eclipse 3.0.1 and improved 
VoIP support. SUSE Pro 9.3 also offers 
improved mobility support for Wi-Fi connec¬ 
tions and Bluetooth devices, PDA and phone 
synchronization; iPod compatability; an inte¬ 
grated firewall, spam blocker and virus scan¬ 
ner; and Novell Evolution 2.0 and Kontact 
3.4. Also included in version 9.3 are the XEN 
virtualization environment and intuitive search 
engines, plus support for AMD Athlon 64 and 
Intel Extended Memory 64 Technology. 

CONTACT Novell Enterprises, 404 Wyman 
Street, Suite 500, Waltham, Massachusetts 
02451, 781-464-8000, www.novell.com. 

SMGateway 


SMGateway is an open-source e-mail/securi- 
ty application from Fortress Systems, Ltd. 
SMGateway offers all of the functionality 
provided by MailScanner and SpamAssassin 
along with extensions and enhancements to 
provide a Web-based interface for users and 
administrators. These added features allow 
administrators to install, control and config¬ 
ure e-mail gateway operations, while allow¬ 
ing users to set their own spam preferences. 
It is designed to provide all e-mail gateway, 
Web access, SQL database, LDAP directory 
and monitoring applications on a single serv¬ 
er. SMGateway features three levels of 
authentication; connectors to Microsoft 
Active Directory, POP- or IMAP-enabled 
directory service; an SQL configuration 
database; LDAP configuration data storage; 
and DCC, Pyzor and Razor2. 

SMGateway is free for customers to down¬ 
load, and Fortress Systems provides three 
levels of support options. 

CONTACT Fortress Systems, Ltd., 3807 
Fulton Street NW, Washington, DC 20007, 
202-338-1670, WWW.fsI.COm, 


OPTion is a virtual thin client for the Linux 
workstation desktop. Compatible with 
GNOME and KDE, it provides a single 
application to connect to all major free and 
commercially available terminal server envi¬ 
ronments. All client sessions are configured 
and managed centrally, and all configured 
client sessions are presented and executed 
from within a central launcher. Client ses¬ 
sions include standard XDMCP, full screen 
and/or within a desktop window; secure 
direct X; secure X login, full screen and/or 
within a desktop window; RDP, full screen 
and/or within a desktop window; xRDP 
with integrated Ericom seamless applica¬ 
tions for WTS 2000/2003 and a cost-free 
RemoteView terminal server agent; ICA with 
server and application browser; Ericom 
PowerTerm Emulator suite; NoMachine NX 



Client, supporting NX Server 1.3 and 1.4; 
and native Tarantella. Supported Linux dis¬ 
tributions include MandrakeLinux, Fedora, 
Novell/SUSE and Xandros. 

CONTACT SmartFLeX Technology, Inc., 

623 Selvaggio Drive, Suite 220, Nazareth, 
Pennsylvania 18064, 610-746-2390, 

www.smartflextech.com 



ConvertX SDK 


Plextor Corporation announced the 
availability of a free Linux software 
developers kit (SDK) for ConvertX 
video-capture devices. The SDK can be 
used to develop for Plextor ConvertX 
PVRs, which offer real-time hardware- 
based MPEG-1, MPEG-2, MPEG-4 and 
Motion JPEG video encoding in a USB 
2.0 video peripheral. The Linux SDK sup¬ 
ports the Video for Linux 2 (V4L2) and 
Advanced Linux Sound Architecture 
(ALSA) specifications. It also supports 
deprecated Open Sound System (OSS) 
applications by way of the OSS compati¬ 
bility layer provided by ALSA. The new 


driver, which requires the Linux 2.6 ker¬ 
nel, includes sample code that can be 
reused in open-source or proprietary appli¬ 
cations to help developers get started. 

CONTACT Plextor America, 48383 Fremont 
Boulevard, Suite 120, Fremont, California 94538, 
510-440-2000, www.plextor.com 


ARCOS 4.0 


Plus Three, LP, released ARCOS 4.0, an application built on Linux, Apache, MySQL and 
Perl and designed to be used by fundraising organizations. Standard features and uses of 
ARCOS are constituent relationship management, e-mail and link tracking, event manage¬ 
ment, social software and an on-line activism center. New features include improved real¬ 
time report generation from databases, an enterprise-class redundancy backup system, and 
a larger and faster user database. ARCOS’ e-mail publishing feature allows users to orga¬ 
nize and distribute e-mail lists based on a variety of factors stored in the database. The 
Web publishing tools offer customizable contributor pages and tell-a-friend pages. In 
addition, the e-mail and Web publishing tools are integrated to allow users to process up 
to two million messages an hour. 

CONTACT Plus Three, LP, 180 Varick Street, Suite #1126, New York, New York 10014, 
212-206-7819, www.plusthree.com B 


Please send information about releases of Linux-related products to Fleather Mead at newproducts@ssc.com or New Products c/o Linux Journal ' PO Box 
55549, Seattle, WA 98155-0549. Submissions are edited for length and content. 
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Why is LPI the Global Standard in 
Linux Certification? 


Trusted. v 

All Linux Professional Institute certification programs are created using extensive 
community input, combined with rigorous psychometric scrutiny and professional 
delivery. We test the whole continuum of important Linux skills - we don't just focus on 
small, subjective tasks. LPI exams are not simply an afterthought used to help sell 
something else. LPI is a non-profit group that does not sell software, training or books. 
Our programs and policies are designed to meet educational requirements, not 
marketing. 

Accessible. 

LPI exams are available in seven languages, at more than 7,000 locations, in more than 
100 countries. You take LPI exams when you want, where you want. In addition, special 
exam lab events around the world make our program even more affordable. And 
because we don't make exclusive partnerships, LPI is supported by a broad range of 
testing centers, book publishers and innovative suppliers of preparation materials. 



You switched to Linux to get away from single-vendor dependence. So why trade one 
form of vendor lock-in for another? LPI's program follows the LSB specification, so 
people who pass our tests can work on all major distributions. Because of its strong 
grass-roots base and corporate support both inside and outside the world of open 
source, LPI goes beyond "vendor-neutral" to truly address community needs. 


LPI is IT certification done RIGHT! 


For more information, please contact us at Linux f 

lnfo@lpi.org or visit us at Professional I 1 

www.lpi.org. Institute 




PHP 5 Power Programming 


by Andi Gutmans, Stig Bakken and Derick Rethans 

Prentice Hall PTR, 2004 I ISBN: 0-131-47149-X I $39.99 US 


PHP, 
arguably 
the world’s 
best Web¬ 
scripting 
language, 
recently 
received a 
significant 
overhaul. 
Version 5 
expands 
the object 
model 

of the language, adds support for new 
MySQL 4.x features and speeds up 
execution. 

However, PHP 4 scripts may not work 
in PHP 5 without some rewriting. PHP 5 
Power Programming is an excellent book 
for PHP 4 developers in need of a PHP 5 
introduction. It’s also a good book for any¬ 


one proficient in another programming 
language, such as Java, Perl or Python, 
who now wants to get started with PHP. 

The book is co-authored by Andi 
Gutmans, Stig Bakken and Derick 
Rethans, three key contributors to the 
PHP language. They bring an intimate 
knowledge of the language to the book 
and provide anecdotal evidence as to 
why PHP has developed in the manner it 
has. Their writing style is clear, focused 
and enjoyable. 

For PHP developers looking for a 
PHP 5 transition guide, this book works 
perfectly. The authors are candid about 
what they’ve broken in the transition from 
PHP 4 to PHP 5. It doesn’t stop there, 
either; coverage of the new PHP 5 object 
model is excellent. Some PHP developers 
may not understand the usefulness of new 
OO concepts introduced in PHP 5, so the 
authors included a chapter on applying 


OO design patterns to PHP. 

PHP and MySQL go together like 
peanut butter and jelly. The improved 
MySQL libraries for PHP further cement 
this relationship. PHP 5 introduces 
native support for SQLite, a powerful 
database option for PHP developers 
without access to another database. 

This book belongs on the desk of 
anyone considering a move to PHP 5. 
It serves as a road map for upgrading 
to the latest incarnation of PHP and as 
a reference for anyone who wants to 
expand his or her PHP object-oriented 
design skills. My copy already has a 
dozen or so sticky notes marking 
important sections and twice as many 
dog-eared pages. It has been an 
invaluable resource in my exploration 
of PHP 5. 

— CHRIS MCAVOY 



Open Source Solutions for Small 
Business Problems 

by John Locke 

Charles River Media, 2004 I ISBN: 1-58450-320-3 I $39.95 US 


Working 
for a num¬ 
ber of 
small busi¬ 
nesses, I 
have seen 
firsthand 
how Linux 
and open- 
source 
software 
can be 
used to 

solve specific problems. It is great to 
see a good book detailing open-source 
solutions for small businesses. John 
Locke takes an excellent approach to 
this subject by addressing both the 
business manager who must decide 


what solutions to implement and the IT 
administrator who must implement 
those solutions. 

Locke covers all of the software you 
need for your small business, including e- 
mail, customer relationship management, 
finance and disaster recovery. Each chapter 
provides valuable background information 
aimed at helping the nontechnical reader 
understand both the problem and the solu¬ 
tion, as well as the details necessary for an 
intermediate Linux or Microsoft Windows 
administrator to implement the solution. 
Locke wisely chooses software that has the 
features you need, as well as strong com¬ 
munity support. He recommends Postfix 
for e-mail because of its security, perfor¬ 
mance and feature set. He also recom¬ 
mends RetrieverCRM for customer rela¬ 


tionship management and SQL-Ledger for 
financial management. Most of the solu¬ 
tions Locke presents will run on Windows 
as well as Linux, for an easy transition into 
the open-source world. 

Although Locke provides good 
instructions on how to implement 
these solutions, there is not enough 
room in his book to provide all of the 
details you may need. For this reason 
he provides many references at the 
end of each chapter, pointing you to 
books, articles and Web sites that can 
provide the details you need. Written 
for a beginning to intermediate user, 
Locke does a great job of keeping the 
chapters simple and easy to follow. 

— STEPHEN HAYWOOD 


OPEN SOURCE 
SOLUTIONS 

for SmAtt Business Problems 
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Knoppix Hacks: 100 Industrial- 
Strength Tips Sr Tools 

by Kyle Rankin 

O’Reilly & Associates, 2004 I ISBN: 0-596-00787-6 I $29.95 US 


As distri¬ 
bution 
innovations 

go, 

Knoppix is 
a revela¬ 
tion. A 
bootable 
CD that 
provides a 
completely 
self-con¬ 
tained and 
fully func¬ 
tional 
desktop? 

All that and it leaves my hard drive 
untouched? What Klaus Knopper has 
wrought with Knoppix is all of that and 
much more. So much more, in fact, that 
even an experienced Knoppix user may 
not have discovered everything the com¬ 
pressed CD offers. 

I received my first Knoppix CD, the 
German edition, from Volker Lendecke. 
Although my limited German language 
facility made sampling that CD a chal¬ 
lenge, I marveled nonetheless as each 
application launched. 

Today, because of the power and 
flexibility of Knoppix, like many other 
people, I burn multiple copies of each 
new release: one for my own use and 
the rest to give away. Just as giving 
away Knoppix CDs fits neatly into my 
advocacy agenda, Knoppix Hacks by 
Kyle Rankin fits into the O’Reilly cata¬ 
log as another excellent book. Rankin 
formally documents what makes 
Knoppix and its derivatives such impor¬ 
tant tools for systems professionals. 

This well-written book offers a 
broad range of descriptions and advice 
about using the capabilities of Knoppix. 
These are presented as a steady progres¬ 
sion of logically grouped hacks. This 
book is a pleasure to read cover to 
cover, but it is as easy to use for an indi¬ 
vidual hack too. 


The range of the hacks presented is 
as impressive as the contents of 
Knoppix itself, including: boot-time 
cheat codes, desktop applications, differ¬ 
ent network-related tools, software 
RAID and troubleshooting. Steps to 
remaster Knoppix to create custom 
derivatives also are discussed. There is 
no unimportant filler. 

The majority of the hacks presented 
is not completely accessible to the 
beginner, but adding the required con¬ 
tent to do so would so encumber this 
book such that it would cease to be use¬ 


ful for the experienced user, who is 
clearly the target for this book. 

If you have not experienced 
Knoppix and cannot download it easily 
for yourself, then by all means let Kyle 
Rankin be your Knoppix-sharing friend. 
Read Knoppix Hacks and explore the 
included Knoppix CD for yourself. If 
you already have experienced Knoppix, 
you should find enough useful hacks 
among the 100 presented in this book to 
warrant its purchase. 


—JEFFREY BIANCHINE 
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Reading File 
Metadata with 
extract and 
libextractor 


Don't just guess about a file's characteristics in a search. Use 
specific extractor plugins to build an accurate database of files. 

BY CHRISTIAN GROTHOFF 


M odern file formats have provisions to annotate 

the contents of the file with descriptive informa¬ 
tion. This development is driven by the need to 
find a better way to organize data than merely 
by using filenames. The problem with such metadata is it is 
not stored in a standardized manner across different file for¬ 
mats. This makes it difficult for format-agnostic tools, such 
as file managers or file-sharing applications, to make use of 
the information. It also results in a plethora of format-specific 
tools used to extract the metadata, such as AVInfo, id3edit, 
jpeginfo and Vocoditor. 

In this article, the libextractor library and the extract tool 
are introduced. The goal of the libextractor Project is to pro¬ 
vide a uniform interface for obtaining metadata from different 
file formats, libextractor currently is used by evidence, the file 
manager for the forthcoming version of Enlightenment, as well 
as for GNUnet, an anonymous, censorship-resistant peer-to- 
peer file-sharing system. The extract tool is a command-line 
interface to the library, libextractor is licensed under the GNU 
General Public License. 

libextractor shares some similarities with the popular file 
tool, which uses the first bytes in a file to guess the MIME 
type, libextractor differs from file in that it tries to obtain 
much more information than the MIME type. Depending on 
the file format, libextractor can obtain additional information, 
including the name of the software used to create the file, the 
author, descriptions, album titles, image dimensions or the 
duration of a movie. 

libextractor achieves this information by using specific 
parser code for many popular formats. The list currently 
includes MP3, Ogg, Real Media, MPEG, RIFF (avi), GIF, 
JPEG, PNG, TIFF, HTML, PDF, PostScript, Zip, 
OpenOffice.org, StarOffice, Microsoft Office, tar, DVI, man, 
Deb, elf, RPM, asf, as well as generic methods such as 
MIME-type detection. Many other formats exist, and among 
the more popular formats only a few proprietary formats are 


not supported. 

Integrating support for new formats is easy, because 
libextractor uses plugins to gather data, libextractor plugins 
are shared libraries that typically provide code to parse one 
particular format. At the end of this article, we demonstrate 
how to integrate support for new formats into the library, 
libextractor gathers the metadata obtained from various plug¬ 
ins and provides clients with a list of pairs, consisting of a 
classification and a character sequence. The classification is 
used to organize the metadata into categories such as title, 
creator, subject and description. 

Installing libextractor and Using extract 

The simplest way to install libextractor is to use one of the 
binary packages available for many distributions. Under 
Debian, the extract tool is in a separate package, extract. 
Headers required to compile other applications against 
libextractor are contained in libextractorO-devel. If you want to 
compile libextractor from source, you need an unusual amount 
of memory: 256MB of system memory is roughly the mini¬ 
mum, as GCC uses about 200MB to compile one of the plug¬ 
ins. Otherwise, compiling by hand follows the usual sequence 
of steps, as shown in Listing 1. 

After installing libextractor, the extract tool can be used to 
obtain metadata from documents. By default, the extract tool 
uses a canonical set of plugins, which consists of all file-format- 
specific plugins supported by the current version of libextractor, 
together with the MIME-type detection plugin. Example output 
for the Linux Journal Web site is shown in Listing 2. 

If you are a user of BibTeX, the option -b is likely to come 
in handy to create BibTeX entries automatically from docu¬ 
ments that have been equipped properly with metadata, as 
shown in Listing 3. 

Another interesting option is -B LANG. This option loads 
one of the language-specific but format-agnostic plugins. These 
plugins attempt to find plain text in a document by matching 
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Listing 1. Compiling libextractor requires about 200MB of memory. 


$ wget http://ovmj.org/libextractor/ 
^download/libextractor-0.4.1.tar.gz 
$ tar xvfz libextractor-0.4.1.tar.gz 
$ cd libextractor-0.4.1 
$ ./configure --prefix=/usr/local 
$ make 

# make install 


Listing 2. Extracting metadata from HTML. 


$ wget -q http://www.linuxjournal.com/ 

$ extract index.html 

description - The Monthly Magazine of the Linux Community 
keywords - linux, linux journal, magazine 


Listing 3. Creating BibTeX entries can be trivial if the documents come with 
plenty of metadata. 


$ wget -q http://www.copyright.gov/legislation/dmca.pdf 
$ extract -b -/dmca.pdf 
% BiBTeX file 
@misc{ unite2001the_d, 

title = "The Digital Millennium Copyright Act 
of 1998", 

author = "United States Copyright Office - jmf", 
note = "digital millennium copyright act 
circumvention technological protection management 
information online service provider liability 
limitation computer maintenance competition 
repair ephemeral recording webcasting distance 
education study vessel hull", 
year = "2001", 
month = "10", 

key = "Copyright Office Summary of the DMCA", 
pages = "18" 

} 


Listing 4. libextractor can sometimes obtain useful information even if the 
format is unknown. 


$ wget -q http://www.bayern.de/HDBG/polges.doc 
S extract -B de polges.doc | head -n 4 
unknown - FEE Politische Geschichte Bayerns 
Herausgegeben vom Haus der Geschichte als Heft 
der zur Geschichte und Kultur Redaktion Manfred 
Bearbeitung Otto Copyright Haus der Geschichte 
Mtinchen Gestaltung furs Internet Rudolf Inhalt im. 
unknown - und das Deutsche Reich, 
unknown - und seine. 

unknown - Henker im Zeitalter von Reformation und Gegenreformation. 


strings in the document against a dictionary. If the need for 
200MB of memory to compile libextractor seems mysterious, 
the answer lies in these plugins. In order to perform a fast dic¬ 
tionary search, a bloomfilter is created that allows fast proba¬ 
bilistic matching; GCC finds the resulting data structure a bit 
hard to swallow. 

The option -B is useful for formats that currently are 
undocumented or unsupported. The printable plugins 
typically print the entire text of the document in order. 
Listing 4 shows the output of extract run on a Microsoft 
Word document. 

This is a rather precise description of the text for a 
German speaker. The supported languages at the moment are 
Danish (da), German (de), English (en), Spanish (es), Italian 
(it) and Norwegian (no). Supporting other languages merely 
is a question of adding free dictionaries in an appropriate 
character set. Further options are described in the extract 
man page; see man 1 extract. 

Using libextractor in Your Projects 

Listing 5 shows the code of a minimalistic program that uses 
libextractor. Compiling minimal.c requires passing the option 
-lextractor to GCC. The EXTRACTOR_KeywordList is a 
simple linked list containing a keyword and a keyword type. 
For details and additional functions for loading plugins and 
manipulating the keyword list, see the libextractor man page, 
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Integrating support for new 
formats is easy, because libextractor 
uses plugins to gather data. 


Listing 5. minimal.c shows the most important libextractor functions in concert. 


#include <extractor.h> 

int main(int argc, char * argv[]) { 

EXTRACTOR_ExtractorList * plugins; 
EXTRACTOR_KeywordList * md_list; 
plugins = EXTRACTOR_loadDefaultLibraries() ; 
md_list = EXTRACTOR_getKeywords(plugins, argv[l]); 
EXTRACTOR_printKeywords(stdout, md_list); 
EXTRACTOR_freeKeywords(md_list); 
EXTRACTOR_removeAll(plugins); /* unload plugins */ 

} 


man 3 libextractor. Java programmers should know that a 
Java class that uses JNI to communicate with libextractor 
also is available. 

Writing Plugins 

The most complicated thing about writing a new plugin for 
libextractor is writing the actual parser for a specific format. 
Nevertheless, the basic pattern is always the same. The plugin 
library must be called libextractor_XXX.so, where XXX 
denotes the file format of the plugin. The library must export a 
method libextractor_XXX_extract, with the following signature 
shown in Listing 6. 


Listing 6. Signature of the function that each libextractor plugin must export. 


struct EXTRACTOR_Keywords * 
libextractor_XXX_extract 
(char * filename, 
char * data, 
size_t size, 

struct EXTRACTOR_Keywords * prev); 


The argument filename specifies the name of the file 
being processed, data is a pointer to the typically mmapped 
contents of the file, and size is the file size. Most plugins do 
not make use of the filename and simply parse data directly, 
starting by verifying that the header of the data matches the 
specific format. 

prev is the list of keywords extracted so far by other 
plugins for the file. The function is expected to return an 
updated list of keywords. If the format does not match the 
expectations of the plugin, prev is returned. Most plugins 
use a function such as addKeyword (Listing 7) to extend 
the list. 

A typical use of addKeyword is to add the MIME type 


Listing 7. The plugins return the metadata using a simple linked list. 


static void addKeyword 

(struct EXTRACTOR_Keywords ** list, 
char * keyword, 

EXTRACTOR_KeywordType type) 

{ 

EXTRACTOR_KeywordList * next; 

next = malloc (si zeof (EXTRACTOR_Keywordl_i st)) ; 

next->next = *list; 

next->keyword = keyword; 

next->keywordType = type; 

*list = next; 

} 


Listing 8. jpegextractor.c adds the MIME type to the list after parsing the file 
header. 


if ( (data[0] != 0xFF) || (data[1] != 0xD8) ) 
return prev; /* not a JPEG */ 
addKeyword(&prev, 

strdup("image/jpeg"), 
EXTRACTOR_MIMETYPE); 

/* ... more parsing code here ... */ 
return prev; 


once the file format has been established. For example, the 
JPEG-extractor (Listing 8) checks the first bytes of the JPEG 
header and then either aborts or claims the file to be a JPEG. 
The strdup in the code is important, because the string will be 
deallocated later, typically in EXTRACTOR_freeKeywords(). 
A list of supported keyword classifications, in the example 
EXTRACTOR_MIMETYPE can be found in the extractor.h 
header file. 

Conclusion 

libextractor is a simple extensible C library for obtaining 
metadata from documents. Its plugin architecture and broad 
support for formats set it apart from format-specific tools. 
The design is limited by the fact that libextractor cannot 
be used to update metadata, which more specialized tools 
typically support. 

Resources for this article: www.linuxjournal.com/article/ 
8207.0 


Christian Grothoff graduated from the University 
of Wuppertal in 2000 with a degree in mathe¬ 
matics. He currently is a PhD student in computer 
science at Purdue University, studying static pro¬ 
gram analysis and secure peer-to-peer network- 
ing. A Linux user since 1995, he has contributed to various free 
software projects and now is the maintainer of GNUnet and a 
member of the core team for libextractor. His home page can 
be found at grothoff.org/christian. 
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Converting 
e-Books to 
Open 
Formats 

E-books are a disappointing flurry of vendor-specific 
formats. Get them converted to HTML to view on 
your choice of device, by marco fioretti 

B ooks in digital format, also known as e-books, can 
be read on devices lacking the power and screen 
space to afford a regular Web browser. Several pub¬ 
lishers, not to mention projects such as Project 
Gutenberg, have provided thousands of new and classic titles 
in digital format. The problem is both the hardware—be it 
generic PDAs or dedicated devices—and the whole e-book 
publishing industry are much more fragmented than are PCs 
and Web browsers. Therefore, it is probable that the e-book 
you recently bought will not be readable ten years from now— 
nor tomorrow, should you decide to use a laptop or change 
PDAs. To help combat this fragmentation, this article discusses 
some existing command-line tools that can convert the most 
popular e-book formats to ASCII or HTML. 

Practically no tools exist now to export e-book formats to 
PDF or OpenDocument, the new OASIS standard used in 
OpenOffice.org, but this is not necessarily a big deal. Once text 
is in ASCII or HTML format, it easily can be moved to plain¬ 
text or PDF format by using a text browser such as w3m or 
programs such as html2ps. If you go this route for conversion, 
you are able to do it today, and because it’s an open format, 20 
years from now too. 

PalmDoc 

On PalmOS, the original and most common e-book format is 
PalmDoc, also called AportisDoc or simply Doc, even though 
it has nothing to do with Microsoft Word’s .doc format. Doc, 
recognizable by the extensions .pdb (Palm Database) or .prc 
(Palm Resource Code), basically is a PalmPilot database com¬ 
posed of records strung together. This standard has spun off 
several variants, including MobiPocket, which adds embedded 
HTML markup tags to the basic format. 

Each Palm e-book is divided into three sections: the header, 
a series of text records and a series of bookmark records. 
Normally, the header is 16 bytes wide. Some Doc readers may 
extend the width at run time to hold additional custom infor¬ 
mation. By default, the header contains data such as the total 
length of the uncompressed text, the position currently viewed 
in the document and an array of two-byte unsigned integers 


giving the uncompressed size of each text record. Usually, the 
maximum size for this kind of record is 4,096 bytes, and each 
one of them is compressed individually. 

The bookmark records are composed of a 16-byte name and 
a 4-byte offset from the beginning of text. Because bookmarks 
are optional, many Doc e-books don’t contain them, and most 
Doc readers support alternative—that is, non-portable—meth¬ 
ods to specify them. Other reader-specific extensions might 
include category, version numbers and links between e-books. 
Almost always, this information is stored outside the .pdb or 
.rc file. Therefore, you should not expect to preserve this kind 
of data when converting your e-books. 

Pyrite Publisher, formerly Doc Toolkit, is a set of content 
conversion tools for the Palm platform. Currently, only some 
text formats can be converted, but functionality can be extend¬ 
ed to support new ones by way of Python plugins. Pyrite 
Publisher can download the documents to convert directly from 
the Web; it also can download set bookmarks directly to the 
output database. The package, which requires Python 2.1 or 
greater, can be used from the command line or through a 
wxWindows-based GUI. The software is available for Linux 
and Windows in both source and binary format. Should you 
choose the latter option, remember that compiled versions 
expect Python to be in /usr. The Linux version can install con¬ 
verted files straight to the PDA using JPilot or pilot-link. 

Pyrite installed and ran flawlessly on Fedora Core 2. Unlike 
the other command-line converters presented below, however, 
Pyrite can save only in ASCII format, not in HTML. The name 
of the executable is pyrpub. The exact command for converting 
.pdb files uses this syntax: 

pyrpub -P TextOutput -o don_quixote.txt \ 

Don_Quixote.pdb 

Pyrite can be enough if all you want to do is quickly index 
a digital library. On the other hand, it is almost trivial to refor¬ 
mat the result to make it more readable in a browser. The snip¬ 
pet of Perl code in Listing 1, albeit ugly, was all it took to pro¬ 
duce the version of Don Quixote shown in Figure 1. 

The script loads the whole ASCII text previously generated 
with Publisher, and every time it finds two new lines in a row, it 
replaces them with HTML paragraph markers. The result then is 
printed to standard output and properly formatted as basic HTML. 
To change justification, fonts and colors, you simply need to paste 
your favourite stylesheet right after the<html><body> line. 

OpenOffice.org 2.0, expected to be released in spring 2005, 
will be able to save text in .pdb format. If it also is able to 
read such files, its mass conversion feature (File—>AutoPilot^ 
Document Converter) would solve the problem nicely. I have tried 
to do this with the 1.9.m65 preview, but all I got was a General 
input/output error pop-up message. Hopefully,this functionali¬ 
ty will be added to future versions. 

The P5 Perl Package 

Pyrite Publisher is designed mainly to go from normal HTML or 
text files to the Palm platform, not the other way around. The 
procedure discussed above is not really scalable to scenarios such 
as converting a great quantity of Palm e-books to customized 
HTML, with hyperlinks and metadata included. In such cases, the 
best solution might be a Perl script combining the standard XML 
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Listing 1. A simple Perl script converts Pyrite's 
extracted text to HTML. 


#! /usr/bin/perl 

undef $/; 

$TEXT = <>; 

STEXT =~ s/\n\n/<p>/gm; 

print <<END_HTML; 
<html><body> 

$TEXT 

</body></html> 

END HTML 



Figure 1. A PalmDoc file converted to HTML for 
viewing in a browser. 

or HTML modules for this language with 
the P5-Palm bundle; these are available 
from the Comprehensive Perl Archive 
Network (see the on-line Resources). The 
P5-Palm set of modules includes classes 
for reading, processing and writing the 
.pdb and .prc database files used by 
PalmOS devices. 

Rocket Ebook and MobiPocket 

RocketBook e-books have several inter¬ 
esting characteristics, including support 
for compressed HTML files and indexes 
containing a summary of paragraph for¬ 
matting and the position of the anchor 
names. These and many more details on 
.rb file internals are explained in the RB 
format page listed in the on-line 
Resources. Rbmake Rocket Ebook and 
Mobipocket files can be disassembled 
with a set of command-line tools called 
Rbmake. Its home page offers source 
code, binary packages, a mailing list and 
contact information to report bugs. To 
use rbmake, you need libxml2, version 
2.3.1 or higher; the pcre (Perl- 
Compatible Regular Expressions) 


library; and zlib, to handle compression. 
To compile from source—at least on 
Fedora Core 2—it also is necessary to 
install the pcre-devel package separately. 

The Rbmake Library 

A nice feature of Rbmake is the source 
code is structured in a modular manner. 

An entire library of object-oriented C rou¬ 
tines can be compiled and linked indepen¬ 
dently from the rest of the package from 
any other program dealing with .rb files. 

In this way, should you want to write your 
own super-customized Rocket Ebook con¬ 
verter or simply index all of your e-books 
into a database, you would need to use 
only the piece that actually knows how to 
read and write the .rb format, the RbFile 
class. This chunk of code opens the file, 
returns a list of the sections composing the 
book and uncompresses on the fly only 
the ones actually required by the main 
program. Should you need them, the 
library also includes functions to match 
and replace parts of the content through 
Perl-compatible regular expressions. 

The Rbmake tools should compile 
quickly and without problems on any 
modem GNU/Linux distribution. 
Exhaustive HTML documentation also is 
included in the source tarball. The binary 
file able to generate HTML files is called 
rbburst. It extracts all the components— 
text, images and an info file—present in 
the original .rb container. Figure 2 shows, 
in two separate Mozilla Windows, the 
cover page and the table of contents of 
the file generated by rbburst when mn on 
The Invisible Man by H. G. Wells. 

Microsoft Reader 

Microsoft’s Reader files, recognizable 
by the .lit extension, have many of the 
characteristics of traditional books, 
including pagination, highlighting and 
notes. They also support keyword 
searching and hyperlinks, but they are 
locked in to one reader platform. 

The tool for converting these files is 
called, simply, Convert Lit. Running the 
program with the -help option lists, 
according to UNIX tradition, all the 
available command-line options. This 
program has three modes of operation: 
explosion, downconversion and inscrib¬ 
ing. Explosion is the one needed to con¬ 
vert an existing .lit file to an OEBPS- 
compliant package. OEBPS (Open 
eBook Publication Structure) is covered 
later in the article. 



Figure 2. Rbmake extracts all the components of a 
RocketBook file, including text and images. 


Figure 3 shows a version of 
Shakespeare’s A Midsummer’s Night 
Dream obtained by using explosion from 
the Convert Lit program. 

Downconversion is the opposite 
process; it generates a .lit file for use by 
a Microsoft Reader-compliant device. 
Inscribing is when the downconversion 
attaches a user-defined label to the .lit 
file. The exact syntax is explained on the 
program’s home page (see Resources). 

We already mentioned that Convert 
Lit creates an OEBPS package made of 
different files. Here is the complete list 
for the example above: Contents .htm, 
copyright .html, ~cov0024.htm, cover.jpg, 
MidSummerNightDream .opf, 

MobMids .html, PCcover.jpg, 
PCthumb.jpg, stylesheet .css and 
thumb.jpg. HTML, CSS and JPG files 
were to be expected, but what is the .opf 
file? It is an XML container describing the 
stmcture and several portions of the origi¬ 
nal book’s metadata. The extension OPF 
stands for open electronic book package 
format. The OPF file contains references 
to the other pieces of the e-book, as well 
as descriptions of their attributes. To have 
a clearer idea of its role, a short excerpt of 
MidSummerNightDream .opf is shown in 
Listing 2. 

The practical consequence of this is 



Figure 3. Convert Lit creates a readable HTML file 
with a hyperlinked table of contents. 
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Listing 2. OPF is an XML-based format for book attributes. 


<dc:Title>A Midsummer-Night's Dream</dc:Title> 
<dc:Creator role="aut" 

file-as="Shakespeare, William, 1564-1616"> 
William Shakespeare, 1564-1616 
</dc:Creator> 

<dc:Description>fiction, poetry</dc:Description> 


Convert Lit could be useful even if you wanted to leave all of your 
collection in a proprietary format. You still could run the program 
on all your .lit e-books and delete everything but the .opf files. 
Then, any quick script or full-blown XML parsing utility could 
scan them and index everything into the database of your choice. 

Convert Lit also removes digital rights management (DRM) 
infections from e-book files using the older DRM1 version. 
And if you have Microsoft Reader e-books, you likely have a 
Microsoft Windows system and a licensed copy of Microsoft 
Reader. According to the Convert Lit Web site, you can build 
and run Convert Lit on Windows to first convert new DRM5 e- 
books to DRM1, using the Windows DRM library. 

Mass Conversion 

In general, we have discussed only com¬ 
mand-line processing in this article. If, 
however, you have a whole collection of e- 
books in different formats, you can convert 
them all at one time with a simple shell 
script. As we already have shown, once the 
text is in ASCII or HTML format, the sky 
is the limit. You can add one or two lines 
to the loop to index with gl i mpse or 
h t: : d i g, print everything in one single 
PostScript book and much more. 

OEBPS 

A solution for putting e-books, at least the 
ones you will be able to get in the near 
future, into an open format is in the works. 

It is the Open eBook Publication Structure 
(OEBPS). Its goal is to provide an XML- 
based specification, based on existing open 
standards, for providing content to multiple 
e-book platforms. OEBPS, which has 
reached version 1.2, is maintained by the 
Open eBook Forum, a group of more than 
85 organizations—hardware and software 
companies, publishers, authors and users— 
involved in electronic publishing. OEBPS 
itself does not directly address DRM. 

However, an OeBF Rights and Rules 
Working Group is studying these issues “to 
provide the electronic publishing commu¬ 
nity with a consistent and mutually sup¬ 
porting set of specifications”. Time will tell 
what will come from this. 

In any case, the open standards on 
which OEBPS is built already are well 


established. Besides XML, Unicode, XHTML and selected parts 
of the CSS1 and CSS2 specifications are represented. Unicode is 
a family of encodings that enables computers to handle without 
ambiguity tens of thousands of characters. XHTML is the refor¬ 
mulation of HTML 4 as XML. In a nutshell, OEBPS could be 
described as nothing more than an e-book optimized extension of 
XHTML—something that won’t go away when some company 
goes out of business. Graphics can be in PNG or JPEG formats. 
Metadata, including author, title, ISBN and so on, will be man¬ 
aged through the Dublin Core vocabulary. 

OEBPS has the potential to preserve all your e-books and 
make sure that the ones you download or buy will not vanish if 
any hardware or software company goes the way of the dodo. 
However, DRM schemes applied on top of these “open” e- 
books still could lock your content in to one vendor. As long as 
you can obtain OEBPS e-books without DRM, OEBPS is the 
best way to guarantee that even if all current e-book hardware 
disappeared, your collection would remain usable. 

Resources for this article: www.linuxjournal.com/article/ 
8208.0 


Marco Fioretti is a hardware systems engineer interest¬ 
ed in free software both as an EDA platform and, as 
the current leader of the RULE Project, as an efficient 
desktop. Marco lives with his family in Rome, Italy 
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One-Click 

Release 

Management 

How a large development project on a tight release 
schedule and a tight budget can use open source to 
tackle the problems of version control and release 
management, by jake davis 


S ay you have a large piece of software, a complicated 
Web site or a whole bunch of little ones. You also 
have a gaggle of coders and a farm of machines on 
which to deploy the end product. Worst of all, the 
client insists on a short turnaround time for critical changes. 
Proprietary products that may provide you with a systematic, 
unified development, testing and deployment process typically 
are expensive and offer limited deployment options. They often 
require new hardware resources and software licenses simply 
to support installation of the system itself. Such a solution can 
be difficult to sell to managers who are concerned about cost 
and to developers who are concerned about learning a new and 
complicated process. 

However, managing the development process from end 
to end on a tight schedule without such a unified approach 
can lead to serious inefficiencies, schedule slippage and, in 
general, big headaches. If you’re the administrator of such 
a project, chances are you’re spending a lot of time dealing 
with the management of code releases. On the other hand, 
you already may be using an expensive piece of proprietary 
software that solves all of your problems today, but the 
higher-ups are balking at the ever-increasing license renew¬ 
al fees. You need to present them with an alternative. It’s 
also possible that you release code only once a year and 
have more free time than you know what to do with, but 
you happen to be an automation junkie. If any of these 
scenarios seem familiar to you, read on. 

The Solution 

Various open-source products can be adapted to minimize costs 
and developer frustration while taming your out-of-control 
release process by serving as the glue between existing 
toolsets. Maybe you even can start making it home in time to 
play a round or two of Scrabble before bedtime. 

I base the examples presented in this article on a few 


assumptions that hopefully are common or generic enough 
that the principles can be extrapolated easily to fit with the 
particulars of a real environment. Our developers probably 
already use a bug-tracking system (BTS), such as Bugzilla, 
ClearQuest or Mantis, or an in-house database solution to 
track change requests and bugs. They also may be using a 
version control system (VCS), such as Arch, CVS or 
Subversion, to manage the actual code changes called for in 
various BTS entries. 

If they’re not using a BTS and a VCS for a large project, 
these developers probably have either superhuman organization 
skills or a high level of tolerance for emotional trauma. Which 
BTS and VCS we use is not essential to this discussion, and 
any exploration of the pros and cons between one system and 
another requires much more text than I am allotted here. In 
short, they all should support the building blocks needed for 
the type of process we’d like to employ. Namely, most any 
BTS can: 

1. Assign a unique ID to all issues or bugs in its database. 

2. Allow you to use the unique ID to track the state of an issue 
and store and retrieve a listing of any source files it affects. 

Any VCS worth its salt (sorry VSS fans) can: 

1. Allow some form of branching and merging of a central 
code hierarchy. 

2. Allow a command-line client process to connect over a 
secure network connection in order to perform updates. 

We use a Subversion (SVN) repository with the SVN+SSH 
access method enabled as our VCS and a generic MySQL 
database table as the BTS. We use Python, which tends to be 
quite readable even for the novice programmer, as our scripting 
language of choice. Chances are your distribution has packages 
for all of these products readily available; configuring them 
will be left as an exercise for the reader. The target machines 
are generic Web servers, all of which support SSH connections 
as well as the VCS client tools. 

Here’s the 10,000-foot overview of the example end-to-end 
process we are likely to be starting out with: 

1. An issue is generated in the BTS and is assigned an ID of 
001 and an initial status of “new”. It includes, or will 
include, a listing of file paths that represent new or changed 
files within the VCS repository and is assigned to the appro¬ 
priate developer. 

2. The assignee developer makes changes to his local copy 
of the source code, checks these changes into the VCS 
repository and updates the status of BTS ID# 001 to 
“in testing”. 

3. The testing server is updated with the new changes. 

4. A QA tester charged with reviewing all BTS items with a 
status of “in testing” verifies that the changes to the code are 
what is desired and updates the status of BTS ID 001 to 
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“ready for production”. 

5. A release manager then packages all changes affected by 

BTS ID# 001 into a release and updates the status of BTS 

ID# 001 to “in production”. 

6. The live server is updated with the changes. 

For the most part, we’re managing to fix bugs and add new 
features to the code base without bugging the system adminis¬ 
trator for much, aside from the occasional password reset or 
RAM upgrade. But steps 3 and 6 require us somehow to get 
the code out of the VCS and onto a live system. We could cut 
and paste files from the VCS into a folder on our hard drive, 
zip it up, send it to the sysadmin and ask him to unzip it on the 
live system. Or, we could take advantage of the structure of our 
VCS and its utilities to do the work for us and completely 
avoid having a conversation with the administrator, whose time 
tends to be a hot commodity. 

The Nuts and Bolts 

If we structured our VCS to encompass a branching scheme 
that mirrors our various statuses in the BTS, we likely 
would end up with a BRANCH to which developers add 
new, untested changes and a TRUNK that includes only 
code that is “in production”, although it easily could be the 
other way around. It then becomes a relatively simple mat¬ 
ter of using the branch merging capabilities of the VCS to 
move “ready for production” code from the testing 
BRANCH to the stable TRUNK. Because no development 
changes happen on our TRUNK, merging from BRANCH 
to TRUNK is not likely to cause any conflicts. Managing 
the last step of moving the code from the VCS to the live 
system becomes even easier, because updating simply is a 
matter of using the VCS client utility to pull down all 
changes that occurred on the TRUNK of the repository. 

So now all the pieces are there to allow quick and accu¬ 
rate code deployment, but we still need to ask our sysadmin 
to run the VCS client tools on the live system. We further 
can minimize our demands on the sysadmin’s time, howev¬ 
er, if he or she is willing to give our release manager an 
SSH login with permission to run the VCS client on the 
live system. 

Expanding the Model to Enable Automated Releases 

Once we’ve got the infrastructure in place to support perform¬ 
ing content updates by way of our VCS, the next logical step is 
to remove further the need for manual intervention at release 
time. It now is possible for us to create a script that can use the 
VCS client tools to pull code updates to a live system. This 
method increases its usefulness as the number of machines we 
need to update increases. If our script has access to a list of all 
the target machines that need to be updated, we can hit them 
all in one fell swoop. 

This piece of the puzzle, like the example, can be a simple 
script that the release manager runs from the command line of 
his workstation. Or, it can be a fancy Web-based GUI that a 
team of release managers can use to update any number of 
machines from any Web browser with a mouse click. In either 
case, it is useful to create a user ID on the client machines that 


Listing 1. vcs_update.py 


#!/usr/bin/env python 
import os, sys 

clientList = ['host1', 'host2', 'hos 13 1 ] 
sandbox = "/usr/local/www" 

def updateClient(client, sandbox): 

# ssh to client machines and update sandbox 
command_line = "ssh %s svn update %s"%(client, 

sandbox) 

output = os.popen4(command_line)[1].readlines() 
for line in output: 
print line 

if_name_=="_main_": 

for client in clientList: 

updateCLient(client, sandbox) 


has permissions to connect back to the VCS system without 
being prompted for login information. This may require config¬ 
uring the user account on the client machines with SSH keys 
that allow it to connect back to the VCS server. 

With this script in place on the client machines, we can 
update client copies of VCS files from a central location over 
an encrypted SSH connection. 

Spreading the Love 

Now we have a reasonably efficient process that piggybacks 
almost seamlessly onto a process that our developers were, for 
the most part, already using. It also allows content updates 
with the click of a button. So what’s stopping us from scripting 
the updates to the testing servers so that they happen automati¬ 
cally at regular intervals, allowing developers the chance to see 
their changes show up on a live test system without asking for 
an update? All we need to do is run the client script on the test¬ 
ing servers as a cron job. 

Also, as long as we’re asking crazy questions, why not 
take advantage of the power of our BTS’ database back end 
to drive the whole process and really cut down on process 
management bottlenecks? To do so, our script generates a list 
of files that need to be merged between branches by running a 
query for all IDs with a status of “ready for production”. The 
script uses the resulting lists as input for functions that perform 
the merge commands and update the BTS ID statuses to “in 
production” automatically. 

Let’s look at our amended 10,000-foot overview now that 
we’ve got all the bells and whistles incorporated: 

1. An issue is generated in the BTS and assigned to the appro¬ 
priate developer. 

2. The assignee developer makes changes to his local copy of 
the source code, checks these changes into the TEST branch 
of the VCS repository and updates the status in the BTS. 
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Listing 2. bts_merge.py 


#!/usr/bin/env python 
import os, MySQLdb 

TRUNK_WC = "/path/to/working_copy_of_trunk/" 
TRUNK_URL = "svn+ssh://vcs-server/project/trunk/" 
BRANCH_URL = " svn+ssh://vcs-server/project/branch/" 

def initDB(): 

# connect to database, return connection cursor 
connection = MySQLdb.connect(host='dbhost', 

db='dbname', 
user='user' , 
passwd='password') 

cursor = connection.cursor() 
return connection, cursor 

def li stllpdatedFi les (cursor): 

# return updated file paths and BTS ids. 

cursor.execute(.SELECT changedfiles 

FROM BugTable 
WHERE status = 

'ready for production.) 

fileList = cursor.fetchall() 

cursor.execute(.'SELECT bugID 

FROM BugTable 
WHERE status = 

'ready_for_production.) 

idList = cursor.fetchall() 
return fileList, idList 


def mergeUpdatedFiles(fileList): 

# merge branch changes into the trunk, 
for fileName in fileList: 

cmd = 'svn merge %s/%s %s/%s '%(BRANCH_URL, 

fileName, 
TRUNKJJRL, 
fileName) 

for line in os.popen4(cmd)[1].readlines(): 
print line 

def updateBTSStatus(idList, cursor): 

# update BTS ids to 'in_production' status, 
for ID in idList: 

cursor.execute(.UPDATE BugTable 

SET status = 'in_production' 
WHERE bugID = %s. % ID) 

def stopDB(connection, cursor): 

# close the database connection 
cursor.close() 

connection.close() 

if_name_=="_main_": 

os.chdir(TRUNK_WC) 
connection, cursor = initDBQ 
fileList, idList = 1istUpdatedFiles(cursor) 
mergeUpdatedFiles(fileList) 
updateBTSStatus(idList, cursor) 
stopDB(connection, cursor) 


3. The testing server content is updated automatically by a 
cron job. 

4. A QA tester verifies that the changes to the code are correct 
and updates the status in the BTS. 

5. A release manager presses a button to launch our merge 
script, which merges all changes into the stable TRUNK and 
updates the BTS. 

6. One last click by the release manager, and the production 
systems are updated to the latest code by way of our VCS 
client script. 

Steps 5 and 6 easily could be combined too, thereby halv¬ 
ing the amount of work our release manager needs to perform. 

Chances are at some point we’ll want to add a staging 
branch to our VCS repository and enable our content update 
system to pull updates from this intermediate branch onto 
a staging server. QA then could see all the changes on a 
live system before the client does. Or, the client could be 
given access in order to provide final approval. Once stag¬ 
ing has been given the thumbs up, moving updates to a 
production system is as easy as performing the already 


automated steps of merging from the staging branch to the 
stable TRUNK and running the content update script against 
the production servers. 

Although these examples represent something of an over¬ 
simplification of the issues involved—for example, we haven’t 
addressed the potential need for database structure updates— 
we have covered some core concepts that can be expanded on 
to build a truly functional, tailor-made system. In fact, we well 
may be approaching development process nirvana, and we still 
haven’t spent dollar one on software licenses. Rather, we’ve 
simply written a few basic scripts to glue together our bug¬ 
tracking and version control systems. As a result, management 
now has more money in the reserve fund and fewer heart palpi¬ 
tations. Our sysadmins have more time to devote to removing 
spyware from desktops. Best of all, we’ve made it home for 
that round of Scrabble with time to spare. That’s the power of 
open source for you. 

Resources for this article: www.linuxjournal.com/article/ 
8141.0 


Jake Davis Oake@imapenguin.com), IT consultant 
and self-described penguin, is cofounder of 
Imapenguin, LLC (www.imapenguin.com) an 
employer of waddling, flightless birds. 
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LOOKE Plug n’ Play * Easy Admin 
Audio Streaming Appliance, LIVE 


GO TO wlSPdirect.com 
CALL 877-881-1954 


SMALL AD. 

BIG PRODUCT! 
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Why I Don't 
Worry about 
SCO, and 
Never Did 

Lawyers can't shut down Linux now. Too many 
important people need it. by chris dibona 


B y the time this article goes to print and arrives in 
your mailbox, the SCO case will mean even less 
than it does when I’m writing this, and that’s saying 
something. The latest headline associated with SCO 
is their potential delisting from the Nasdaq for failing to file 
paperwork. This is the public-company equivalent of being 
sent to bed without supper or being expelled from school. It 
isn’t good, and it’s very irresponsible. 

By the time this magazine hits print they’ll have sent in 
their homework, late, for a lesser grade, or they’ll have 
retreated from the public markets and the expensive and 
revealing Sarbanes-Oxley scrutiny that comes with having 
a ticker symbol. Either way, they’ll be even less of a threat 
to free software, but I have to say, I wasn’t worried, not for 
one minute. 

I wasn’t worried about their legal position that they owned 
parts of the Linux kernel. 

I wasn’t worried about their complaints against friend of 
Linux, IBM. 

I wasn’t worried about the future of the Linux kernel, Linus 
himself, his wife, his kids, Alan Cox, Andrew Morton or, for 
that matter, the people in industry that SCO subpoenaed in pur¬ 
suit of their action against IBM. 

Why wasn’t I worried? The time to sue Linux and many 
prominent open-source software projects has passed, and in 
that passing, we have a blueprint on how to avoid consequen¬ 
tial litigation for future important free software projects. The 
reason I don’t worry about people suing Linux, and the reason 
I wasn’t worried when SCO did it, is because Linux has 
become too important to too many people for it to be vulnera¬ 
ble to that kind of attack. 

The time to kill Linux was when it was a project with ten 
developers who lived on university stipends, not when it has 
thousands of connected developers and $14 billion in Linux- 


related sales (IDC’s number for the year 2003, if you believe 
analysts). It was vulnerable when it was still a university pro¬ 
ject, not now when uncountable school districts are using it to 
reduce their dependence on the punitive cost structures of pro¬ 
prietary software. It was vulnerable when it was in use in a few 
countries by a few dozen users, not now when it is used by a 
few dozen countries to ensure their software sovereignty. In 
short, it was vulnerable when it meant nothing to a few, not 
now when it is central to the Information Age economies. 

And if that hyperbole didn’t make you turn the page and 
drool over an ad for some sexy cluster gear, here is what we 
learn from Linux and Litigation. 

Lirst, if you want to destroy a free software project’s 
chances of success, start when it is young, by making a 
competing product so good there will be no user need for 
the free software. 

Second, if you are running a large project, assemble your 
friends around you, the way Linux has, so you can absorb the 
hits that come with success. Surrounding Linux is a vast array 
of industry organizations, corporate users, nations and end 
users whose livelihoods are tied tightly to Linux. This doesn’t 
mean that Linux doesn’t get sued, it simply means the people 
doing the suing find themselves horribly alone. 

Third, put off any sort of foundation or corporatization until 
you are ready to defeat the slings and arrows that come with 
the success of your project. The Samba team has played this 
card very well. If some large software company were to go 
after Samba under the rubric of patent infringement, who could 
it go after that would slow the use of Samba? Samba Project 
leaders Andrew Tridgell and Jeremy Allison would be protect¬ 
ed by the same companies who find Samba vital to their sur¬ 
vival. And this wouldn’t stop people from using Samba one bit. 
Sometimes in the pages of LJ we talk about Microsoft as if it 
were filled with fools. But it’s not so foolish as to sue its end 
users the way SCO has. The day for effectively suing Samba 
also has passed. 

And finally, when people sue Linux or Samba, help if 
you can, but the best start is to keep using free software 
and help keep it vital to yourself, your workplace and your 
government. Through this need, this necessity, we enshroud 
our software with a warm blanket of security that the para¬ 
sites will fail to penetrate. 

So, in the end, this article is really about Asterisk, the free 
Voice-over-IP software that developed at telco hardware maker 
Digium. Asterisk represents a project that is on the verge of 
being too important to be vulnerable. If I were looking for an 
open-source company to back, I’d say (and have said) Digium. 
If you haven’t tried it, you really should. It is remarkable stuff. 
Like Linux, you can’t believe software can be this good and 
useful. And if it’s good and useful, it will be important to 
enough people that legal threats from failed companies just 
won’t matter.@ 


Chris DiBona is the Open Source Program Manager for Mountain 
View, California-based Google, Inc. Before joining Google, Mr 
DiBona was an editor/author for the popular on-line Web site 
slashdot.org. He writes for a great number of publications, speaks 
internationally on software development and digital rights issues 
and co-edited the award-winning essay compilation Open 
Sources , whose seguel is planned for a July 2005 release. 
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User Management 

- support more than 3000 PPPoE 
or Hotspot clients 

- full radius support for 

user parameters 

- tx/rx speed, address, 

filter rules 

- supports radius real time 

modification of parameters 
while users are online 

- Peer to peer control (P2P) 

- burst time 

- per client P2P tx/rx rules 

- P2P pool 

- complete blocking of P2P 


Wireless 

and 


backbone 


Wireless monitoring 

- Frequency scanning with 
detailed report 

- Raw wireless packet sniffer 

- streaming option to 

Ethereal analyzer 

- option to save to a 

file format supported 
by Ethereal 

- Snooper packet inspection 

- analyzes all raw frames 

received for wireless 
parameters 

- monitor a single channel 

or all channels 



RouterBOARD 44 RouterBOARD 230 

For the Router Builder ■ No feature left behind I 


- rackmount servers and routers 

- up to 24 Ethernet ports in a PC 

- no more straight/cross cable 

problems 

- server quality VIA VT6105 chips 


Integrated router with various interfaces. 
Use as an AP on a tower with up to 
500ft PoE, Includes lDE/CF r miniPCI, 
USB r PCMCIA, UART, PCI, GPIO, 

LCD controller, Linux SDK, and more. 



RouterBOARD 11/14/13 

Multi radio tower I 



$240 


- PoE 

- 10-56V input 

- 9x 10/100 

- 6x mPCI 


RouterBOARD 500 & RouterBOARD 564 


The Wireless Switchboard I 


MiniPCI to PCI adapters for multi 
radio system. Tested with sixteen 
radios in one Router/AP 


For a complete multi-radio tower system, 
the RouterBOARD 500 can carry a 
daughterboard (RouterBOARD 564) which 
adds six ethernets and four miniPCI. 


- Nstreme wireless polling protocol 

* no decrease in speed 
over long distances 
(as seen with the 802.11 
ack packet bottleneck) 

- polling improves speed 
and eliminates contention 
for access to the 
wireless bandwidth 

- access point control over 

Nstreme clients tx data 
to optimize use of the 
wireless medium 

- radius support for the 
access control fist 
including bandwidth 
settings for wireless clients 

- Full 802.1 la/b/g support 

The above is a brief 
description of a few features, 
for more information and a 
fully featured 24 hour demo 
go to: 



RouterBOARD 500 


$140 

(kiw vDtlage version) 


- Linux Board Support Package 

(full Debian MIPS installation) 

- 2G6-4Q0Mh2 MIPS CPU 

- 64MB NAND storage 

- Compact Flash 

- 32MB DDR 

- Low power 

- PoE 802,3af 

standard and 
passive PoE 
(also 12V PoE) 

- 10^24V and 25-48V 
power mode 

- 3 10/100 Ethernets MDI-X 

- 2 miniPCI (one on each side) 

- 2-3x faster for networking 

than the Geode SC1100 boards 

- 2Q0-3Q0MB/S aggregate throughput 

- L3 RouterOS license Included 


contact sales@routerboard.com or go to www.routerboard.com 



www.mikrotik.com 






Microway® Quad Opteron™ Cluster with 
36 Opteron 852s, redundant power and 
45 hard drives in CoolRak™ cabinet. 
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MPI Link-Checker 


to the Rescue! 


A single slow node or intermittent link can cut the speed of MPI applications by half. 
Whether you use GigE, Myrinet, Quadrix, InfiniBand or InfiniPath HTX, there is only 
one choice for monitoring and debugging your cluster of SMP nodes: 

Microway's MPI Link-Checker™ 

Our unique diagnostic tool uses an end-to-end stress test to find problems with 
cables, processors, BIOS's, PCI buses, NIC's, switches, and even MPI itself! The 
newest release provides ancillary data on inter-process and intra-CPU latency 
which can vary by a factor of 10 between MPI versions. MPI Link-Checker is 
also useful for porting applications to new hardware. It provides instant details 
on how latency and bandwidth vary with packet size. It is available now for a free 
30 day evaluation! 

Wondering what's wrong with your cluster, or need help designing your next one? 
Call our HPC staff at 508-746-7341. Visit microway.com to learn about new low 
latency interconnects including the PathScale InfiniPath HTX Adapter, which 
delivers unmatched MPI latency of under 1.5 microseconds. 

Microway has been an innovator in HPC since 1982. We have 
thousands of happy customers. Isn't it time you became one? 


Call us first at 508-746-7341 for quotes and benchmarking 
services. Find technical information, testimonials, and 
newsletter at www.microway.com. 



TMaoway 

23 Years of Expertise Built In 


PathScale 


Atcelfi rating 
Cluster Performance" 






















































